ludwig — an Introduction
Last week, I was wondering if we could have the power to just take our dataset and treat it like a fuel, feed it into a big fat machine (of course, metaphorically) which needs data as fuel to run and then sit back and relax while all the work is done by that machine. Even though there are some tools available but nothing satisfied me in terms of what I hoped.
Surprisingly, yesterday I stumbled into ludwig, an open source tool by Uber (launched around 3 weeks back) which allows us to have that machine I wished.
All you have to do is get your csv file ready with the data, write two lines of command and “Voila!” , ludwig starts and trains ur data. It not just trains the data, it also gives option to visualise your accuracy and track the stats. It is built on tensorflow and thus allows to use your gpu as well.
With a little bit of knowledge in Deep Learning models, you can even customise your models without writing any code. With a stronger knowledge, you can quickly setup your own complex custom model. It is really useful to obtain strong baselines (even sophisticated models such as Image Captioning) for comparison and also for experimenting with different base models to get an idea of how different models behave on our data.
Internally, ludwig has encoders for all data types be it texts, categories, numerical, binary, images etc. It selects the encoder according to our input datatypes which converts them into corresponding tensors. It also has a combiner which combines all the input encoders, processes them and returns the tensors for output decoders. Finally the decoders use this data to map it to output features, whatever we have provided.
It selects its own training hyperparameters, splits data into training, validation and test. The only slight pain that you have to take is define what type of data you are feeding it which is defined in the .yaml file or it can be in the form of json if you are using python API.
The installation and usage is pretty much simple and straightforward which is well explained in https://uber.github.io/ludwig/getting_started/
To get a look at basic examples — https://uber.github.io/ludwig/examples/
ludwig repo — https://github.com/uber/ludwig
