Panic when deleting file with a non-UTF-8 name on a Mac

[eugenesvk]

Because it converts all paths to strings for some reason during the "canonicalization" step

trash-rs/src/macos.rs

Line 65 in 1a0fc59

let full_paths = full_paths.into_iter().map(to_string).collect::<Result<Vec<_>, _>>()?;

[Byron]

Thanks a lot pointing this out!

It should definitely not be needed to do that, and I'd hope that a fix can be contributed.

[eugenesvk]

Upd: think it's just a matter of escaping binary chars to match the url format spec

Do you know much about NSStrings and → NSURLs that is fed into the file manager's trashItemAtURL_resultingItemURL_error apple's call?

I've tried to fix it, but think there is no FileManager API that can delete an arbitrary bag of bytes that OsString can represent, it only accepts ~dozen encodings.
But then if I use an encoding-aware API stringWithCString_encoding to decode OsString (converted to CString to feed into this API), I get the wrong file deleted - the one with the utf-8 name, not the one with the alternative encoding (I guess bytes+encoding results in a utf8 URL?).

The rust's non-encoding-aware deprecated stringWithCString api doesn't produce the needed NSString, but AnyObject

(also tried with other 3 brew Swift (uses FileManager) /objc/py cli trash tools, and they also bug with non-utf8, though doesn't neccessarily mean there is no way)

At the very least we could check (in Rust) whether the passed path is utf-8 and report and error if it's not instead of crashing, but it's a surprising limitation

[Byron]

Upd: think it's just a matter of escaping binary chars to match the url format spec

I see, so applying a URL encoding to the bytes would do the trick? I wouldn't be surprised if NSURL is what's passed to the file-manager.

[eugenesvk]

It should, but the problem is I can't find an objc API for such a basic thing:

1. NSURL should get created from fileURLWithPath(path: &NSString), so needs an NSString
   • there is a way to create it from bytes URLWithDataRepresentation_relativeToURL, but it autoconverts bytes to their utf8 codes and escapes them and can't match the file name: for a file with a single \xf8 (U+00F8 codepoint or utf8 0xC3 xB8 or ø) file name that in Finder is seen as %F8 (so, %-byte-escaped), this api results in %C3%B8 name, which is utf8
2. But then if we use NSString as input, which API allows to create an NSString from bytes with proper auto-escaping of invalid bytes?
   • there is an api to add %-encoding stringByAddingPercentEncodingWithAllowedCharacters, but that works on NSStrings, which we don't have yet...

I've even found another objc crate that has a method to get NSString from OsStr, but that one also failed with invalid utf8 src

(yes, FM gets NSURL

trash-rs/src/macos.rs

Lines 80 to 84 in 1a0fc59

	let url = unsafe { NSURL::fileURLWithPath(&string) };
	trace!("Finished fileURLWithPath");
	trace!("Calling trashItemAtURL");
	let res = unsafe { file_mgr.trashItemAtURL_resultingItemURL_error(&url, None) };

)

[eugenesvk]

@madsmtm you're the resident Objc guru that added objc2 crate. Would you please suggest how to do this simple thing:

• take an arbitrary bag of bytes file path (OsStr in Rust) and
• create an NSUrl in objc2 that can be fed to the FileManager for further processing?

[madsmtm]

Haven't researched closely yet, but what about -[NSURL fileURLWithFileSystemRepresentation:isDirectory:relativeToURL:]?

[madsmtm]

Where you'd get the pointer with OsStrExt to get the bytes + reallocating and adding a NUL byte to that with CString.

[eugenesvk]

That was the first one I've tried, also thought that the FS repr would mean raw bytes, but I get a panic

unexpected NULL returned from -[NSFileManager stringWithFileSystemRepresentation:length:]

snippet I used for testing

let in_oss = OsStr::from_bytes(b"/Volumes/Untitled/\xf8");
let bpath = in_oss.as_bytes();
let cstring = CString::new(bpath).expect("CString::new failed to create from given path");
let cstring_len = cstring.count_bytes();
println!("cstring: {:?}={cstring_len}",cstring);
let nncstring = NonNull::new(cstring.into_raw()).expect("REASON");
println!("nncstring: {:?}",nncstring);
let string:Retained<NSString> = unsafe {file_mgr.stringWithFileSystemRepresentation_length(nncstring, cstring_len)};

this works with utf-8 let in_oss = OsStr::from_bytes(b"/Volumes/Untitled/x");, though, so I also assumed this was an encoding error. But maybe there is a simpler mistake?

[madsmtm]

No I mean fileURLWithFileSystemRepresentation:isDirectory:relativeToURL: on NSURL, not stringWithFileSystemRepresentation:length: on NSFileManager

[eugenesvk]

Sorry about the confusion, too many tests :)
Well, this one requires an unnecessary "is_dir" check, though might've also checked it before, but checked now as well, also can't find the file even though debug-prints the URL as %F8 (in the correct version from valid utf8 it's %25F8, so % is escaped as well)

// URL from CString via fileURLWithFileSystemRepresentation_isDirectory_relativeToURL
  // FAILS with binary path
  let path_b:&[u8] = b"/Volumes/Untitled/\xf8";
  let path_os:&OsStr = OsStr::from_bytes(&path_b);
  let path_b:&[u8] = path_os.as_bytes();
  // OK with valid utf8 manual %-encoding
  let path_os:&OsStr = OsStr::new("/Volumes/Untitled/%F8");
  let path_b:&[u8] = path_os.as_bytes();

  // OK (different valid utf-8 string)
  let path_os:&OsStr = OsStr::new("/Volumes/Untitled/ピ");
  let path_b:&[u8] = path_os.as_bytes();


  // same in all cases
  let mut p:PathBuf = PathBuf::new(); p.push(path_os); let path_p = p;
  if path_p.is_file() {println!("path_os is a file! {:?}",path_os);}
  let cstring = CString::new(path_b).expect("CString::new failed to create from given path");
  let cstring_len = cstring.count_bytes();
  println!("pre={:?}\nc{}={:?}",&path_os,cstring_len,&cstring);
  let nncstring = NonNull::new(cstring.into_raw()).expect("REASON");
  let is_dir = false;
  let url_s = unsafe{NSURL::fileURLWithFileSystemRepresentation_isDirectory_relativeToURL(nncstring,is_dir,None)};
  // println!("fileURLWithFileSystemRepresentation_isDirectory_relativeToURL\n{url:?}",);

  let res_s = unsafe{file_mgr.trashItemAtURL_resultingItemURL_error(&url_s, None) };//
  let err_s = match res_s {Ok(())=>format!("ok"), Err(err)=>format!("{err}"),};
  println!("fileURLWithFileSystemRepresentation_isDirectory_relativeToURL:\nurl_s={url_s:?}\n\nres_s={err_s}");

I've meanwhile adapted the std's from_utf8_lossy function, but instead of replacing invalid utf8 byte sequences with a replacement char append them in a percent encoding, seems to work.