Garbage in, garbage out is outdated

I was at a government event recently and someone, I think from DEFRA, asked a question I hear a lot. Their point, which is a fair point, is that they struggle with having patchy records and missing data. Missing address data, whatever it is in their system. They've got quite patchy data and that limits the amount and the quality of analysis they can do with it.

His question was: is it still the case that it's garbage in, garbage out with the AI tools you set up on top?

Some people's view is yes, that is still the case, you need good quality data at the start. Actually, with modern tooling, and there are a few different ways of doing this, you can improve and enrich data in a rapid way.

What used to happen

Previously you'd maybe have to go through manually, record by record, and go "no, that postcode isn't in that region, that can't be right," and either fix it or ignore that record.

I've done exactly that use case on mapping functions. If you had garbage data in the background and tried to run AI models on top of it, the model would just go "we don't recognize this locality," so ignore it. You might have a thousand customers in one town or one region, or a million customers, but you're only going to log the ones it recognizes.

That's kind of fine if it's consistently inconsistent. But some things are going to be more inconsistent than others. You might think you've got a higher proportion of your customers in a particular region just because it's easier to populate that region well with data, when actually you're skewing results because other places are harder to spell or have longer names. It's a Venn diagram of all the different spellings of a place merged together. The same thing happens with SKU data.

Garbage in, good out

Now you can do all of that automatically. Snowflake has tools like AI Classify. Say you've got London as the city. Some people might have "City of London," some people might have "LDN," some people might have "Lundun" or something like that. You can put AI Classify on top of that and it will reclassify all those records into the same output.

And it will check that with you. If you're using something like Kleene MCP, you can run that and it will check with you and go "okay, we've got 50 different versions that we think are actually the same thing. Do you want us to apply that across all of them?" So you can do it at scale, but there's still a human in the loop. It's not just going to randomly reassign or recategorize things.

Functions like that actually mean garbage in, good out.

Where you fix it

There are a few places you could fix it. Obviously, if possible, you'd want to go back into the original record. Often that's not possible. You're not going to change the original record. So what you'll normally do is pull in the raw data, and from that point onwards you keep the original record. You might have city or locality, but then you create extra versions of that, something like "locality recategorized." So you have an enriched version and you use that downstream.

What you might also want to do eventually is reverse ETL on that, which is taking the updated record and pushing it back into the source system. Something like an ERP like NetSuite, for your customers or your suppliers. You might eventually push back in and go "there was this erroneous version of that record, we can push it back in." That's going to help improve processes like delivery, if you're an e-commerce business, by making sure you've got the right record at source where your delivery system pulls from. Ultimately you have the golden record in your source system rather than it just sitting in a data warehouse with data tools downstream.

We're already doing this for our clients. Maybe not this exact case every time, but something similar. So if a company previously thought "we've got bad data, we couldn't make the most of these models because it wouldn't reconcile the data together," it's not a problem anymore.

The other tropes on their way out

It's not just data quality issues like that. It's also data matching. If you're trying to match customer records to create a full customer journey, or you're trying to match business supplier records, they might have "Limited" in full in one system and "LTD" in another. We've got fuzzy matching tools in the app that can bring those together so you can create golden records like that as well.

Data duplication is another one. If you've got two customer records in your source system and they look almost identical, but one has a slightly different email address or a slightly different postcode, historically it's actually quite hard, even with Snowflake and modern data warehousing, to compress them down to one version unless you go in and build specific rules. So at the minute you'd be reporting two customers rather than one. You might be sending multiple emails, multiple physical marketing campaigns, all that sort of stuff out, which is just inefficient.

Snowflake has another tool for this that can look at complete customer records. It's not just looking at one field or two fields, it can look at a hundred fields and go "we think this is actually just one record," and then you can use that to compress it down. So it's not necessarily individual data quality on the record. It's whether the overall set of records you have is the right one, or whether it should be adjusted and compressed a little bit.

If your team has written off AI analytics because the data isn't clean enough, speak to the Kleene.ai team.