Change charset for all tables.

alex — Mon, 24 Sep 2007 13:31:04 +0000

A lot of linux distributions comes with mysql with default charset latin1. In modern world it seems to be rudimentary, for i18n utf8 is a must. So converting existing mysql databases/tables to utf8 is not an uncommon task. There’s no problem with it, mysql has ALTER TABLE SET CHARSET statement. But what if you have several dozens of tables? And what if each table has a lot of text fields? You’ll spend a lot of time in front of mysql console or phpmyadmin. Or you’ll have to generate script from table/field names.
I offer simple solution when all your data is english (I can’t predict behaviour if you have other data).

Make mysqldump of your entire database
Change all charset definitions sed -i 's/latin1/utf8/' dumpfile.sql
Load dump back to database

Parsing large mysql dumps

alex — Sat, 07 Jul 2007 21:22:50 +0000

Recently I faced a problem – I had large mysql dump on remote machine and I needed just some tables from it. Dump file was about 900 Mb, bzip’ed – around 60 Mb. I didn’t want to download it just for about 1 Mb of required data. First idea that came into my head – load this dump to the database on remote machine and then dump again only required tables. The problem that I had no possibility to do that – I had just ftp/ssh access. So, I opened this ~1Gb sql file with vi. It opened it even with syntax highlighting! But when I tried to remove 2 millions of unnecessary lines it hanged forever… Emacs wasn’t better.

I had no wish to learn something like sed, so I wrote small perl script for this task. Usage is simple:

perl strip-dump.pl your-dump-file.sql table1 table2 table3

Script source:

#!/usr/bin/perl -w
#Get only required tables from the whole dump
use strict;
 
my $dump_file = shift @ARGV;
my $out_dump_file = 'stripped_dump.sql';
 
open DUMP, "<$dump_file" or die $!;
open OUT, ">$out_dump_file" or die $!;
 
my @required_tables = @ARGV;
 
my $table_dump_started = 0;
while ()
{
	if (index($_, 'CREATE TABLE') >= 0)
	{
		$table_dump_started = 0;
		foreach my $table (@required_tables)
		{
			if (index($_, "`$table`") >= 0)
			{
				$table_dump_started = 1;
				last;
			}
		}
	}
	print OUT
		if $table_dump_started;
}
 
close OUT;
close DUMP;

OXT blog » mysql

Change charset for all tables.

Parsing large mysql dumps