Understanding M17n

Posted about 1 year ago in Character Encodings.

Big changes are coming to Ruby in version 1.9 with regard to character encodings. Ruby is going from a language with some of the weakest character encoding support to arguably some of the best support out there for working with different encodings. We're all grown up now.

The downside is that the new code comes with a good size learning curve. I would know because I recently battled through figuring it out so I could add support to the standard CSV library for nearly all of the encodings. It was a battle too. It's brave new territory and there's not a lot of help out there yet for understanding Ruby's new features.

I'm hoping to change that.

This posting will be the start of a new series of blog articles designed to explain the character encoding support in Ruby 1.9. I'm going to assume you know absolutely nothing about character encodings though and begin by explaining in detail what they are and why we have them.

After that, we're going to examine the character encoding support in Ruby 1.8. There's a lot less support there to examine, but it's not well understood and I'm hoping that seeing it in detail will help with understanding how and why Ruby 1.9 is changing.

Finally, we will examine all the new encoding features of Ruby 1.9 in as much detail as possible. We will literally cover it all. Along the way, I'll talk strategy and give you all the helpful tips I know to successfully managing character encodings, in general as well as with Ruby specifics.

This message will serve as a table of contents for this series of posts, so you may want to bookmark it if this topic is of interest to you. Here are all of the posts, in order:

  1. What is a character encoding?
  2. The Unicode Character Set and Encodings
  3. General Encoding Strategies
  4. Bytes and Characters in Ruby 1.8
  5. The $KCODE Variable and jcode Library
  6. Encoding Conversion With iconv
  7. Ruby 1.8 Character Encoding Flaws
  8. Ruby 1.9's String
  9. Ruby 1.9's Three Default Encodings
  10. Miscellaneous M17n Details
  11. What Ruby 1.9 Gives Us
LUcas EFe added about 3 hours later:

Thank you, man. This is really appreciated.

Regards

Lucas

Csiszár Attila added 2 months later:

Very informative and nice articles!

Could you give some insights about date localization too? I struggled last time with localizing dates and I'm not happy at all with Ruby's capabilities. This is an another weak point in 1.8x and I'm curious how this will be changing in 1.9?

Thanks, Attila

James Edward Gray II added 2 months later:

Ruby itself doesn't really provide any localization tools. You will need to look into add-on libraries for this.

The latest versions of Rails have some support for this, but it's pretty young and unrefined, in my opinion. For full featured support you might want to try a library like Ruby-GetText-Package.

roger added 9 months later:

Fascinatingly, googling for m17n returned a link to this in the top 10 :)

James Edward Gray II added 9 months later:

Here's another good resource for m17n documentation by Brain Candler: http://github.com/candlerb/string19/tree/master.

James Edward Gray II added 9 months later:

When you are ready to deal with character encodings from Ruby's C API, Yugui has some tips for that.

Joe Grossberg added about 1 year later:

Just a quick suggestion: it would be nice if you stated that M17N == "multilingualization", for those new to the subject and unfamiliar with the jargon.

James Edward Gray II added about 1 year later:

I do mention it when I get into coverage of 1.9's system, but you're right that I probably should have said it on this introduction.

Rohan D added about 1 year later:

Sexy article

Add Your Thoughts

You can use Markdown in the body of your comment to format text and make links.

Note that I reserve the right to edit any content you post here. I typically exercise this right to fix formatting issues. All posts must be approved so spam will never be seen on these pages.

Author:
URL or Email (optional):
Body: