[Mahout] Deploying custom drivers to mahout

Developing custom drivers on Mahout is fairly straightforward. You can inherit from MahoutDriver for Java drivers and MahourSparkDriver for spark drivers.

The Javadoc for MahoutDriver (if you can find it) provides a good summary of how to implement it

General-purpose driver class for Mahout programs. Utilizes org.apache.hadoop.util.ProgramDriver to run main methods of other classes, but first loads up default properties from a properties file.

To run locally:

$MAHOUT_HOME/bin/mahout run shortJobName [over-ride ops]

Works like this: by default, the file “driver.classes.props” is loaded from the classpath, which defines a mapping between short names like “vectordump” and fully qualified class names. The format of driver.classes.props is like so:

fully.qualified.class.name = shortJobName : descriptive string

The default properties to be applied to the program run is pulled out of, by default, “.props” (also off of the classpath).

The format of the default properties files is as follows:

  i|input = /path/to/my/input
  o|output = /path/to/my/output
  m|jarFile = /path/to/jarFile
  # etc - each line is shortArg|longArg = value

The next argument to the Driver is supposed to be the short name of the class to be run (as defined in the driver.classes.props file).

Then the class which will be run will have it’s main called with

main(new String[] { "--input", "/path/to/my/input", "--output", "/path/to/my/output" });

After all the “default” properties are loaded from the file, any further command-line arguments are taken in, and over-ride the defaults.

So if your driver.classes.props looks like so:

org.apache.mahout.utils.vectors.VectorDumper = vecDump : dump vectors from a sequence file

and you have a file core/src/main/resources/vecDump.props which looks like

  o|output = /tmp/vectorOut
  s|seqFile = /my/vector/sequenceFile

And you execute the command-line:

$MAHOUT_HOME/bin/mahout run vecDump -s /my/otherVector/sequenceFile

Then org.apache.mahout.utils.vectors.VectorDumper.main() will be called with arguments:

You can also implement it slightly differently by just dumping the jar into the mahout home directory and naming it starting with “mahout-” i.e. mahout-mydriver.jar

Hope this helps someone

Setting up Mahout in Linux

A few simple steps to get Mahout running in Linux. This is mostly about the bash script to get it to run easily

You’ll need to install Java first, then download and unpack the mahout distribution.

I then placed it in /usr/local/mahout

To be able to run Mahout from the path, the following bash script was placed in /usr/local/bin

Update the paths as relevant

export MAHOUT_JAVA_HOME=/usr/lib/jvm/java-8-oracle/jre/
export MAHOUT_HOME=/usr/local/mahout

[UE4] Unreal Engine & miniupnp

This post covers how to integrate pnp into an unreal project using miniupnp.

The first step is to clone the project from github

The module that we are interested in is the miniupnpc and in that directory, there is another directory called msvc and this contains the solution file for Visual Studio. Open this and if you have a more recent version of visual studio (which very likely do), it will want to upgrade everything. Let it go through the upgrade process.

Building the project now will most likely fail due to a missing file miniupnpcstrings.h . This file needs to be generated and the way to do that is to run a script in that folder called updateminiupnpstrings.sh. You will most probably need something like cygwin to for this script to work as it a unix shell to work

Once the miniupnpcstrings.h has been generated, we also need to follow some instructions for Unreal Engine for Linking Static Libraries Using The Build System, particularly the section on customizations for targeting UE4 modules.

From the project properties page, choose configuration manager. From the Active Solutions Platform, select new and type in or select x64 and save it. You have to do this for only one of the projects.

Building of the static project will fail since it can’t find the lib which is now in x64\Release as opposed to just Release\. The exe is not required for integrating with Unreal Engine, but if you want to complete the build, just fix the path in Project Properties -> Linker -> Input.

You should choose the release build instead of the debug build and you should now be able to build the solution from visual studio. It did pop up some warnings for me, but the build completed successfully.

The rest of the instructions are from the unreal engine documentation about integrating static libraries, starting from section about Third Party Directory

[UE4] Source Control

As with any software project, it is important to use some form of source control solution. For most software projects, there are a number of good solutions out there. However, in the world of game development, most of these are not viable since they won’t handle binary files very well, and unreal engine (as most games) will have a large amount of binary resources.

Perforce is a good tool for small teams since it is free for teams with less than 20 developers.

Another thing that can be confusing is what files/folders to add into the source control. Generally, we do not want to include any files which can be auto-generated (e.g. builds) or are transient (e.g. logs). You should generally include the following folders into source control

  • Config
  • Content
  • Intermediate/ProjectFiles/<projectname>.vcxproj* (the .vcxproj.user file may not be relevant if there are multiple developers)
  • Source
  • <projectname>.sln
  • <projectname>/.uproject

I found it odd that the project file is in the intermediate folder since one wouldn’t intuitively think to include it in source control

Unending Love

I seem to have loved you in numberless forms, numberless times…

In life after life, in age after age, forever,
My spellbound heart has made and remade the necklace of songs,

That you take as a gift, to wear around your neck in many forms

The Joy of Misery

I want to write of happy things,

joyous ocassions,

of which there sure are aplenty…

yet, for some some reason,

perhaps a long standing conditioning,

I keep getting dragged back,

back into the mud,

into the quicksand,

a murky swamp…


Why oh why?

why is there so much joy,

in the misery an sorrow?

in the chewing bubble gum,

to solve a maths equation.


Why oh why,

do I,

am I,

do I,

with little to carry around,

worry that I indeed have little,

i have only this trouble,

this trouble or that…


I am lacking,

not in joy,

yes in joy,

for I am lacking,

in troubles…

[UE4] Intellisense Errors and USTRUCT

The Intellisense feature of Visual Studio, while useful is quite different from the compiler and on occassion produces false positive errors. According to the document about Setting Up Visual Studio for UE4, this can happen for few possible reason

  • The IntelliSense compiler (EDG) is more strict than the MSVC compiler.

  • Some #defines are setup differently for IntelliSense than when building normally.

  • C++ compiled by IntelliSense is always treated as 32-bit.

However, there is also the case where the Intellisense is just plain wrong. USTRUCT() structures are just such a case. You will find that Intellisense complains about definitions with the structures defined as USTRUCT.


This may not be a big deal, except for the fact these fales positives makes it difficult to spot actual errors and I like to see a clean workspace with no errors (not even false positives). Fortunately, there is a workaround. You can include the USTRUCT definitions only if it is not being compiled by Intellisense.


Fortunately, it is just the GENERATED_USTRUCT_BODY() line that is the culprit. To get these errors to go away, instead of that single line, use the following

	#ifndef __INTELLISENSE__ /* Eliminating erroneous Intellisense Squigglies */ 

and that’ll make those pesky squiggly lines to disappear and give you a nice clean workspace 😀


[UE4] Getting Sprint to work (C++)

Getting Shift to sprint in the Unreal Engine 4 is very easy. There are technically two ways of doing it.

  • Change the scale down to say 0.5 and then increase it to 1.0 for sprinting
  • Change the MaxWalkSpeed for walking and sprinting.

I prefer the second technique because it will keep the code simple enough and continue to function as expected with analog controllers.

Lets start by adding the input action bindings

Edit -> Project Settings -> Input -> Bindings and add an action for Sprint with Left Shift for the key.

All the code is in the Character class.

Within the header file, there are two blueprint variables for the WalkSpeed and SprintSpeed so they can be modified easily.

We also override the BeginPlay() Method. This is used to default to Walking.

Add the following to your character header file

	/** The speed at which the character will be walking */
	UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = Character)
	uint16 WalkSpeed;

	/** The speed at which the character will be sprinting*/
	UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = Character)
	uint16 SprintSpeed;

	virtual void BeginPlay();

	/** Enables Sprinting for the character*/
	void EnableSprint();

	/** Disables Sprinting again */
	void DisableSprint();

Add the following to the constructor so we have defaults

AMyCharacter::AMyCharacter(const FObjectInitializer& ObjectInitializer)
	: Super(ObjectInitializer)

    // Other code
	// Set the default Defaults 😉
	WalkSpeed = 250;
	SprintSpeed = 600;


Plug in the Sprint action to the methods by adding the following lines into the SetupPlayerInputComponent method:

	InputComponent->BindAction("Sprint", IE_Pressed, this, &AMyCharacter::EnableSprint);
	InputComponent->BindAction("Sprint", IE_Released, this, &AMyCharacter::DisableSprint);

And implement the rest of the methods

void AVSCharacter::BeginPlay()

    // Ensure the player starts with Walking

void AVSCharacter::EnableSprint()
	CharacterMovement->MaxWalkSpeed = SprintSpeed;
	GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Black, TEXT("Sprintin"));

void AVSCharacter::DisableSprint()
	CharacterMovement->MaxWalkSpeed = WalkSpeed;
	GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Black, TEXT("Walkin"));

[UE4] Getting Multiplayer Player Pawn AI Navigation to work (C++)

Unreal Engine is an awesome piece of technology making it easy to do almost anything you might want.

When using the Top Down view however, there is a hurdle to get over when trying to get multiplayer to work. This is a C++ project solution to this problem based on a BluePrints solution.

The basic problem stems from the fact that

SimpleMoveToLocation was never intended to be used in a network environment. It’s simple after all 😉 Currently there’s no dedicated engine way of making player pawn follow a path. ” (from the same page)

To be able to get a working version of SimpleMoveToLocation, we need to do the following:

  • Create a proxy player class (BP_WarriorProxy is BP solution)
  • Set the proxy class as the default player controller class
  • Move the camera to the proxy (Since the actual player class is on the server, we can’t put a camera on it to display on the client)

The BP solution talks about four classes – our counterparts are as follows:

  • BP_WarriorProxy: ADemoPlayerProxy
  • BP_WarriorController: ADemoPlayerController (Auto-created when creating a c++ top down project)
  • BP_Warrior: ADemoCharacter (Auto-created when creating a C++ top down project)
  • BP_WarriorAI: AAIController – we have no reason to subclass it.

So, from a standard c++ top down project, the only class we need to add is the ADemoPlayerProxy – so go ahead and do that first.

The first thing we’ll do is rewire the ADemoGameMode class to initialise the proxy class instead of the default MyCharacter Blueprint.


ADemoGameMode::ADemoGameMode(const class FPostConstructInitializeProperties&amp; PCIP) : Super(PCIP) 
    // use our custom PlayerController class 
    PlayerControllerClass = ADemoPlayerController::StaticClass(); 

    // set default pawn class to our Blueprinted character 
    /* static ConstructorHelpers::FClassFinder<apawn> PlayerPawnBPClass(TEXT("/Game/Blueprints/MyCharacter")); 
    if (PlayerPawnBPClass.Class != NULL) 
        DefaultPawnClass = PlayerPawnBPClass.Class; 

    DefaultPawnClass = ADemoPlayerProxy::StaticClass(); } 


Our Player Proxy class declaration

/* This class will work as a proxy on the client - tracking the movements of the 
 * real Character on the server side and sending back controls. */ 
UCLASS() class Demo_API ADemoPlayerProxy : public APawn 
    /** Top down camera */ 
    UPROPERTY(VisibleAnywhere, BlueprintReadOnly, Category = Camera) TSubobjectPtr<class ucameracomponent=""> TopDownCameraComponent; 

    /** Camera boom positioning the camera above the character */ 
    UPROPERTY(VisibleAnywhere, BlueprintReadOnly, Category = Camera) TSubobjectPtr<class uspringarmcomponent=""> CameraBoom; 

    // Needed so we can pick up the class in the constructor and spawn it elsewhere 
    TSubclassOf<aactor> CharacterClass; 

    // Pointer to the actual character. We replicate it so we know its location for the camera on the client 
    UPROPERTY(Replicated) ADemoCharacter* Character; 

    // The AI Controller we will use to auto-navigate the player 
    AAIController* PlayerAI; 

    // We spawn the real player character and other such elements here 
    virtual void BeginPlay() override; 

    // Use do keep this actor in sync with the real one 
    void Tick(float DeltaTime); 

    // Used by the controller to get moving to work 
    void MoveToLocation(const ADemoPlayerController* controller, const FVector&amp; vector); 

and the definition:

#include "Demo.h"
#include "DemoCharacter.h"
#include "AIController.h"
#include "DemoPlayerProxy.h"
#include "UnrealNetwork.h"

ADemoPlayerProxy::ADemoPlayerProxy(const class FPostConstructInitializeProperties&amp; PCIP)
: Super(PCIP)
	// Don't rotate character to camera direction
	bUseControllerRotationPitch = false;
	bUseControllerRotationYaw = false;
	bUseControllerRotationRoll = false;

	// It seems that without a RootComponent, we can't place the Actual Character easily
	TSubobjectPtr&lt;UCapsuleComponent&gt; TouchCapsule = PCIP.CreateDefaultSubobject<ucapsulecomponent>(this, TEXT("dummy"));
	TouchCapsule-&gt;InitCapsuleSize(1.0f, 1.0f);
	RootComponent = TouchCapsule;

	// Create a camera boom...
	CameraBoom = PCIP.CreateDefaultSubobject&lt;USpringArmComponent&gt;(this, TEXT("CameraBoom"));
	CameraBoom-&gt;bAbsoluteRotation = true; // Don't want arm to rotate when character does
	CameraBoom-&gt;TargetArmLength = 800.f;
	CameraBoom-&gt;RelativeRotation = FRotator(-60.f, 0.f, 0.f);
	CameraBoom-&gt;bDoCollisionTest = false; // Don't want to pull camera in when it collides with level

	// Create a camera...
	TopDownCameraComponent = PCIP.CreateDefaultSubobject&lt;UCameraComponent&gt;(this, TEXT("TopDownCamera"));
	TopDownCameraComponent-&gt;AttachTo(CameraBoom, USpringArmComponent::SocketName);
	TopDownCameraComponent-&gt;bUseControllerViewRotation = false; // Camera does not rotate relative to arm

	if (Role == ROLE_Authority)
		static ConstructorHelpers::FObjectFinder&lt;UClass&gt; PlayerPawnBPClass(TEXT("/Game/Blueprints/MyCharacter.MyCharacter_C"));
		CharacterClass = PlayerPawnBPClass.Object;


void ADemoPlayerProxy::BeginPlay()
	if (Role == ROLE_Authority)
		// Get current location of the Player Proxy
		FVector Location =  GetActorLocation();
		FRotator Rotation = GetActorRotation();

		FActorSpawnParameters SpawnParams;
		SpawnParams.Owner = this;
		SpawnParams.Instigator = Instigator;
		SpawnParams.bNoCollisionFail = true;

		// Spawn the actual player character at the same location as the Proxy
		Character = Cast&lt;ADemoCharacter&gt;(GetWorld()-&gt;SpawnActor(CharacterClass, &amp;Location, &amp;Rotation, SpawnParams));

		// We use the PlayerAI to control the Player Character for Navigation
		PlayerAI = GetWorld()-&gt;SpawnActor&lt;AAIController&gt;(GetActorLocation(), GetActorRotation());


void ADemoPlayerProxy::Tick(float DeltaTime)

	if (Character)
		// Keep the Proxy in sync with the real character
		FTransform CharTransform = Character-&gt;GetTransform();
		FTransform MyTransform = GetTransform();

		FTransform Transform;
		Transform.LerpTranslationScale3D(CharTransform, MyTransform, ScalarRegister(0.5f));


void ADemoPlayerProxy::MoveToLocation(const ADemoPlayerController* controller, const FVector&amp; DestLocation)
	/** Looks easy - doesn't it.
	 *  What this does is to engage the AI to pathfind.
	 *  The AI will then "route" the character correctly.
	 *  The Proxy (and with it the camera), on each tick, moves to the location of the real character
	 *  And thus, we get the illusion of moving with the Player Character

void ADemoPlayerProxy::GetLifetimeReplicatedProps(TArray&lt; class FLifetimeProperty &gt; &amp; OutLifetimeProps) const


	// Replicate to Everyone
	DOREPLIFETIME(ADemoPlayerProxy, Character);

We’ll now cover changes to the Player Controller. We’ll rewire it here to ask the proxy to move, which will in turn ask the AIController to find a path and move the real player character. 

This involves changing the SetMoveDestination method to call a server side method to move the character. When the character moves, the player Proxy will automatically mirror the movements.

In the header file, add the following function

    /** Navigate player to the given world location (Server Version) */
    UFUNCTION(reliable, server, WithValidation)
    void ServerSetNewMoveDestination(const FVector DestLocation);

Now let’s implement it (DemoPlayerController.cpp):

bool ADemoPlayerController::ServerSetNewMoveDestination_Validate(const FVector DestLocation)
	return true;

/* Actual implementation of the ServerSetMoveDestination method */
void ADemoPlayerController::ServerSetNewMoveDestination_Implementation(const FVector DestLocation)
	ADemoPlayerProxy* Pawn = Cast<ademoplayerproxy>(GetPawn());
	if (Pawn)
		UNavigationSystem* const NaDemoys = GetWorld()-&gt;GetNavigationSystem();
		float const Distance = FVector::Dist(DestLocation, Pawn-&gt;GetActorLocation());

		// We need to issue move command only if far enough in order for walk animation to play correctly
		if (NaDemoys &amp;&amp; (Distance &gt; 120.0f))
			//NaDemoys-&gt;SimpleMoveToLocation(this, DestLocation);
			Pawn-&gt;MoveToLocation(this, DestLocation);


And finally, the rewiring of the original method:

void ADemoPlayerController::SetNewMoveDestination(const FVector DestLocation)


Finally, in terms of the character class, the only change is really to remove the camera components that we moved to the Player Proxy which I shall leave to you :-p

Frostfall & DayZ

When it comes to gaming, my preferences seem to lie in the RPG Genre with Strategy and Simulators coming in second. It is not often that I dabble in First Person shooters. While I have some fond memories of the multiplayer mods that came out of Half Life, I did not find myself curious about the features of Battlefield 4.

The features of DayZ, on the other hand are interesting. While it doesn’t really have any RPG elements, it does seem to have unusual features and when the mod came out, it was noticed for being different… and difficult…

My brother has been trying to get me to play DayZ for a little while and I did try the mod a while back. I found it to be a bit “too real” to be really enjoyable. From the near explosion of similarly themed games recently, it does seem like a popular genre though…

Continue reading