TOCB: Data Migration Tool For Couchbase

When I was working on my pet project, I had to copy data from MySQL database to Couchbase. I searched the internet to find a solution. I found couple scripts that would copy all MySQL tables into Couchbase. But I didn’t want just to copy tables. The Couchbase bucket structure in my mind was different than normalized sql tables.

Then a thought came to my mind that what if we move data by sql queries. You write a query in sql and the results will be copied to Couchbase.

So, I decided to write a small tool to implement that idea. The result: tocb.

Using tocb is very simple. You don’t need Go language knowledge except you’ll have to install the language. You can refer to README section of the repository about how to use the tool here: https://github.com/chapani/tocb

You will need some TOML knowledge. Fortunately, it’s very simple. You can find detailed information about TOML here: https://github.com/toml-lang/toml

This tool is extensible as well. Just copy mysql.go to your databases name and modify it. Don’t forget to change “handler” property in your config.toml. Also, add another condition to tocb.go: https://github.com/chapani/tocb/blob/master/tocb.go#L41-L48

In future I’d like to add these features:

  1. Field casting (string, integer, float, boolean).
  2. Ability to migrate huge data in smaller steps using OFFSET and LIMIT.
  3. Ability to rollback if something undesired happens.
  4. Support for other popular databases like Postgres, Sqlite.

I hope someone will find it useful. Feel free to hit me with suggestions or bug reports in Github Issues page for the repo. (https://github.com/chapani/tocb/issues)

CategoriesGo

Leave a Reply

Your email address will not be published. Required fields are marked *