Pfthb

apologies if this is a really stupid question or already asked elsewhere. I'm reading in some JSON and using decode_json on it, then extracting text from it and outputting that to a file.

My problem is that Unicode characters are encoded as eg u2019 in the JSON, decode_json appears to convert this to x2019. When I grab this text and output to a UTF8-encoded file, it appears as garbage.

Sample code:

use warnings;
use strict;
use JSON qw( decode_json );
use Data::Dumper;

open IN, $file or die;
binmode IN, ":utf8";
my $data = <IN>;
my $json = decode_json( $data );
open OUT, ">$outfile" or die;
binmode OUT, ":utf8";
binmode STDOUT, ":utf8";
foreach my $textdat (@ $json->'results' ) 
 print STDOUT Dumper($textdat);
 my $text = $textdat->'text';
 print OUT "$textn";

The Dumper output shows that the u encoding has been converted to x encoding. What am I doing wrong?

decode_json needs UTF-8 encoded input, so use from_json instead that accepts unicode:

my $json = from_json($data);

Another option would be to encode the data yourself:

use Encode;

my $encoded_data = encode('UTF-8', $data);
...
my $json = decode_json($data);

But it makes little sense to encode data just to decode it.

decode_json expects UTF-8, but you're passing decoded text (Unicode Code Points) instead.

So, you could remove the existing character decoding.

use feature qw( say );
use open 'std', ':encoding(UTF-8)';
use JSON qw( decode_json );

my $json_utf8 = do 
 open(my $fh, '<:raw', $in_qfn)
 or die("Can't open "$in_qfn": $!n");

 local $/;
 <$fh>;
 ;

my $data = decode_json($json_utf8);


 open(my $fh, '>', $out_qfn)
 or die("Can't create "$out_qfn": $!n");

 for my $result (@ $data->results ) 
 say $fh $result->text;
 

Or, you could use from_json (or JSON->new->decode) instead of decode_json.

use feature qw( say );
use open 'std', ':encoding(UTF-8)';
use JSON qw( from_json ); # <---

my $json_ucp = do 
 open(my $fh, '<', $in_qfn) # <---
 or die("Can't open "$in_qfn": $!n");

 local $/;
 <$fh>;
 ;

my $data = from_json($json_ucp); # <---


 open(my $fh, '>', $out_qfn)
 or die("Can't create "$out_qfn": $!n");

 for my $result (@ $data->results ) 
 say $fh $result->text;
 

The arrows point to the three minor differences between the two snippets.

I made a number of cleanups.

• Missing local $/; in case there are line breaks in the JSON.


• Don't use 2-arg open.


• Don't needlessly use global variables.


• Use better names for variables. $data and $json were notably reversed, and $file didn't contain a file.


• Limit the scope of your variables, especially if they use up system resources (e.g. file handles).


• Use :encoding(UTF-8) (the standard encoding) instead of :encoding(utf8) (an encoding only used by Perl). :utf8 is even worse as it uses the internal encoding rather than the standard one, and it can lead to corrupt scalars if provided bad input.


• Get rid of the noisy quotes around identifiers used as hash keys.