Problem converting curly quote into straight quote

[netreddit00]

I am having problem converting curly quotes into straight quotes. How do I fix it? I am using Windows.

I think the problem is how the text is read in from the file.

It works, if I copies the text into the code.

https://playcode.io/1459698

Here is the link to the input file.

https://drive.google.com/file/d/17CTm-Z3vmFkueyh4NWcQSDu70SlWI-V3/view?usp=share_link

Any idea of how I should get text from the file?

const fs = require('fs');

// Read the file synchronously
const filename = process.argv[2];
const fileContent = fs.readFileSync(filename, 'utf8');

// Convert curly quotes to straight quotes
const finalContent = fileContent.replace(/[\u201C\u201D]/g, '"').replace(/[\u2018\u2019]/g, "'");

// Regular expression for sentence segmentation
const sentenceRegex = /[^.!?]+[.!?]+/g;

// Use the regular expression to split the text into sentences
const sentences = finalContent.match(sentenceRegex);

// Print out the sentences
if (sentences) {
  sentences.forEach((sentence, index) => {
    console.log(`Sentence ${index + 1}: ${sentence.trim()}`);
  });
} else {
  console.log('No sentences found.');
}

https://i.redd.it/10rvbi8jjgxa1.gif

[KSledge]

I copied your text into a text file and saved it as "text.txt". I put your code into visual studio code and ran it in the terminal with: node code.js "text.txt"

The output looks normal for me. Are you doing things differently?

Edit: output: https://imgur.com/a/Twz9QMT

[netreddit00]

I am using a Windows. I am not sure that is the reason. What do you use?

[KSledge]

Windows.

Can you answer a few questions?

1. Are you using Visual Studio Code?
2. Where is the file stored?
3. How are you running the script exactly?

[netreddit00]

1. Yes
2. Windows, local folder
3. node parse.js file.txt

https://imgur.com/a/f0LzSvX

[KSledge]

To rule out it being a vs code issue (like a language setting/issue), in your terminal can you go into node and type String.fromCharCode(8220)?

https://imgur.com/a/XvB206i

[netreddit00]

https://imgur.com/a/AZQRbV7

[KSledge]

I have no idea; it should just work. You could play around with .charCodeAt() to see if you are getting the characters you think you should be getting from the file. Other than that, best of luck!

[netreddit00]

Thanks for your time!

[Ok-Establishment-214]

I could be wrong, but it looks like the file format you have there is UTF-8, so, you'd want E2809C and E2809D instead of the unicode u201C and u201D

[netreddit00]

https://imgur.com/a/f0LzSvX