kandan kandan - 10 months ago 72
MySQL Question

Perl Remove invalid characters, invalid latin1 characters from string

I have a perl script that reads from a web service and saves in a mysql table. this table uses latin1. from the web service there are coming some wrong characters and need to remove them before saving them in the database, otherwise they get saved as '?'

wanted to do something similar as:

but is not removing them.

the webservice that has the wrong characters is: https://jobvacancies.services.businesslink.gov.uk:8443/vacancy/26653478

using a user agent to get the data, seems coming in utf8 but the characters need to be removed:

my $ua = LWP::UserAgent->new ();

$ua->default_headers->push_header ('Accept' =>
"text/html,application/xhtml" .
$ua->default_headers->push_header ('Accept-Charset' => "utf-8");

my $doc = $ua->get ("https://jobvacancies.services.businesslink.gov.uk:8443/vacancy/26653478")


If you just want to remove the characters outside the 7-bit ascii set (which are sufficient to display messages in english), you can you do this: