Challenges in the evaluation of machine learning techniques in generative urban design

Evaluating the quality of generative design machine learning tools is a critical challenge. Existing methods range from human-based assessments to performance-based metrics and statistical comparisons. We focus on generative urban design and critically review the evaluation methods employed in recent literature. We experimentally test and comprehensively analyze these methods. We find that existing approaches favor disrupted designs over well-designed ones, have inherent limitations, and fail to capture the tool performance quality. To address this critical gap, we develop two strategies: (1) modifying the Fréchet Inception Distance (FID) score to align with specific design principles, and (2) leveraging visual language models to assess design outputs. Our experiments show that these approaches provide more robust and comprehensive evaluations. The findings underscore the need for practical and reliable evaluation frameworks for AI in design fields to advance the research in this field.