The workspace to check in to is not an ArcSDE workspace. Likely you have specified the incorrect geodatabase for the check-in workspace. The check-in-to workspace should point to the geodatabase from which data has been checked out.
Ensure you have specified the correct geodatabase.
Ensure that the checkout replica you have specified is listed and that the replica role is listed as Parent. If it is listed as Child, you have incorrectly selected the checkout workspace.