Serverless PDI in AWS – Building the jar

So, following on from the first post in this series, here’s all the technical gubbins.

Firstly, how do you build PDI as an engine?  Well simple – you need to create a pom.xml and use maven.

The key parts of that file are:

  1. Adding the Pentaho repository
  2. Defining pentaho.kettle.version
  3. Adding the core lambda java libraries
  4. Figuring out that the vfs library version needs to be this weird thing: 20050307052300
  5. And then the key point – using the “Maven shade” plugin, which basically gathers up the whole lot and dumps it into a jar suitable for uploading directly to AWS.

What next? Well topics for next few weeks include:

  • The java code wrapper to launch PDI
  • Logging/Monitoring
  • Triggering
  • Persistence (S3 / redshift)
Advertisements

One thought on “Serverless PDI in AWS – Building the jar

  1. Hi Dan, I really like this approach of a lean kettle engine triggered by a lambda. Would you be able to point me in the direction of some background reading to understand how to use the kettle engine to load jobs/transforms programatically as part of a java project in general? Searching for this kinda stuff is tricky as terms like “integration” and “engine” are overloaded all over the place. I hope you can help.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s