After observing, the door is closed, but the window on the second floor is still open. If you jump up, you can probably enter it.
The cloth shop looks quite large, with a large plaque hanging on the doorframe, which says a few big words about the Silk Shop...